The Japan Football Association president said the country is still preparing to host the Tokyo 2020 Olympics, now postponed until next year because of the coronavirus.
"I sensed resistance against hosting two top events in the same country in a short period of time, which was reinforced by the impact of the coronavirus pandemic," Kozo Tajima said in a statement.
He also noted the ASEAN Football Federation had already swung towards a joint bid from Australia and New Zealand, and that Japan had hosted several FIFA tournaments in recent years.
Japan's withdrawal leaves the event a two-way race between Colombia and the joint ANZAC bid, which is regarded as the favourite.
The decision by Japan comes weeks after Brazil announced it would withdraw from the process, citing belt-tightening forced by the coronavirus.
The Brazilian Football Confederation said it was struggling to secure the private and public funding needed to mount a competitive bid, and was throwing its weight behind Colombia.
FIFA is due to announce its decision on the winning bid on June 25, with the field having been whittled down from an original 10.
Women's football -- and female sport in general -- is gaining popularity, with the last women's World Cup in France watched by 1.12 billion people globally.
The next will have 32 teams for the first time, up from 24.
Japan won praise for its hosting of last year's Rugby World Cup -- the first time the tournament has been played in Asia.
Despite being interrupted by a deadly typhoon, the tournament was viewed as a success, drawing massive crowds and stirring domestic enthusiasm for rugby -- especially after the Brave Blossoms won a historic quarter-final spot.
The postponed Summer Games are scheduled to open on July 23, 2021, but experts have expressed doubts about whether the coronavirus pandemic will be under control by then.
